Method for predicting tracking cameras for free-roaming mobile robots

ABSTRACT

In a manufacturing environment, a controllable mobile apparatus such as a mobile robot is guided through the manufacturing environment by a navigation system that utilizes fixed overhead cameras having a predefined viewing area. As the robot leaves a first viewing area there is an uncertainty as to which viewing area the robot will go into so that that camera may be monitored. The uncertainty is minimized by using a plurality of rings of uncertainty which overlap the surrounding areas. The uncertainty is continuously eliminating by drawing the circle of uncertainty into smaller and smaller area.

This invention is related to the following pending U.S. patentapplications, assigned to Texas Instruments Incorporated, and all filedon Aug. 30, 1985, which by reference are incorporated herein, Ser. Nos.:771,431; 771,443; 771,322; 771,379; 771,432; 771,433 (allowed); 772,280;771,329; 771,442; 771,397; 772,061; 771,380; and 771,545.

BACKGROUND AND SUMMARY OF THE INVENTION Background of the Invention

This invention relates to controllable mobile apparatuses such as mobilerobots and more particularly to controllable mobile apparatuses thatutilize a video system for navigation and still more particularly tocontrollable mobile apparatuses that utilize a video system fornavigation and include a plurality of fixed mounted video cameras with apredefined field of view of an area of coverage.

When navigating controllable mobile apparatuses through a factory ormanufacturing environment using fixed overhead video cameras, aparticular controllable mobile apparatus will periodically leave theviewing area or zone coverage of one video camera and enter the zonecoverage of a second video camera. The mobile apparatus's path cannot bepredicted because the mobile apparatus is free roaming and its path isvariable. In the past, mobile apparatuses have not been free roaming andthus were limited to tracks, rails or fixed guide paths which make theirmotion very predictable. Freedom is unpredictability of the robot'smotion gives rise to the problem of local navigation sensor predictions,i.e., TV cameras.

Summary of the Invention

In a manufacturing environment, a controllable mobile apparatus such asa mobile robot is guided through the manufacturing environment by anavigation system that utilizes fixed overhead cameras having apredefined viewing area. As the robot leaves a first viewing area thereis an uncertainty as to which viewing area the robot will go into sothat the camera may be monitored. The uncertainty is minimized by usinga plurality of rings of uncertainty which overlap the surrounding areas.The uncertainty is continuously eliminating by drawing the circle ofuncertainty into smaller and smaller areas.

It is an object of the invention to provide a free roaming mobileapparatus which is guided by fixed overhead cameras in which the fieldof view of the fixed overhead cameras do not overlap and the freeroaming mobile apparatus can transfer between one field of view to thenext field of view. The system has an ability to predict which overheadcameras field of view the controllable mobile apparatus is entering.

These and other objects and advantages of the inventin will be apparentfrom reading of the specification in conjunction with the figures inwhich:

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a manufacturing system according to theinvention:

FIG. 2 is a block diagram of the control and navigation circuit of FIG.1;

FIG. 3 is is a block diagram of the interface module of FIG. 2;

FIG. 4 is a block diagram of the visual histrogram processor of FIG. 2;

FIG. 5 is a block diagram of the window generator of FIG. 4;

FIG. 6 is a block diagram of the picture memory module of FIG. 2;

FIGS. 7A-7F are waveform diagrams illustrating the operation of thecontrol and navigation circuit 15 of FIG. 1;

FIGS. 8A-8C are flow diagrams of the centroidification of the imagesaccording to the invention;

FIG. 9 is an isometric illustration of a manufacturing facilityaccording to the invention;

FIGS. 10 and 11 illustrate the problem encountered with reflectiveimages;

FIG. 12 is a flow diagram of the program implemented when reflectedimages are encountered;

FIGS. 13, 14 and 15 indicate the problem encountered when a guidancebeacon is obstructed from view of a camera according to the invention;

FIGS. 16A-16B, and 17 illustrates the determination of an obstructedimage;

FIG. 18 is a block diagram of a mobile robot according to the invention;

FIG. 19 is a block diagram of a servo control loop according to theinvention;

FIG. 20 is a block diagram of a dead reckoning navigation systemaccording to the invention;

FIG. 21A-21C are flow diagrams of the dead reckoning navigation systemimplemented by the mobile apparatus of FIG. 18;

FIG. 22 is a flow diagram of the execution of a servo filter accordingto the invention;

FIGS. 23, 24, 25A-25B, and 26A-26B illustrate a braking system for themobile apparatus of FIG. 18;

FIG. 27 is a block diagram of a wireless communication system forcommunicating between the control and navigation circuits and the mobileapparatus;

FIG. 28 is a waveform of the wireless communication system of FIG. 27;

FIG. 29 is a top view of a beacon according to the invention;

FIG. 30 is a sectional view of a navigation beacon according to theinvention;

FIG. 31 i s a top view of a manufacturing facility according to theinvention;

FIG. 32 is an isometric view of the manufacturing facility of FIG. 31;

FIG. 33A-33B is a flow diagram of a program used to determine theposition of a mobile apparatus when it is out of the view of the cameraaccording to the invention;

FIG. 34 is a block diagram of a manufacturing facility having a multiapparatus system according to the invention;

FIG. 35 is a flow diagram of the operator interface of FIG. 34;

FIG. 36 is a flow diagram of the communication logic of FIG. 34;

FIG. 37 is a flow diagram illustrating the operation of the navigationlogic of FIG. 34;

FIG. 38 is a flow diagram illustrating the operation of the factorycontroller of FIG. 34;

FIG. 39 is a flow diagram illustrating the operation of the plannerscheduler of FIG. 34;

FIG. 40 is a flow diagram illustrating the operation of the batterystation logic of FIG. 34;

FIGS. 41A-41G provide a flow diagram of the control of a mobileapparatus system such as that illustrated in FIG. 34;

FIGS. 42 through 49G are figures which illustrate the generation ofrules for a multimode mobile apparatus system;

FIGS. 50A-50C through 52 illustrate a method of controlling themovements of a mobile apparatus in a multiple mode factory environment;

FIGS. 53A-53B are flow diagrams illustrating the operation of a mulltimobile apparatus system;

FIGS. 54A-54B through 56 are flow diagrams illustrating the navigationand control of multiple apparatus using a single control and navigationcircuit 15;

FIGS. 57 through 60 provide a collision avoidance system for themultiple apparatus according to the invention; and

FIGS. 61A-61D are the table of equations.

DETAILED DESCRIPTION OF THE EMBODIMENTS

In FIG. 1, to which reference should now be made, there is shown asimplified diagram of a control and navigation system 2A and mobileapparatus 21 according to the invention. The control and navigationsystem 2A includes a plurality of cameras 1, control and navigationcircuit 15, a communication system 19 and a terminal 17. Each member ofthe plurality of cameras 1 in the illustrated embodiment are raster scanTV cameras and converts light that is emitted from beacons 3, 5 and 7 tovideo signals. Becons 3, 5 and 7 are mounted on the mobile apparatus 21and in the embodiment of FIG. 1 are light-emitting diodes operating inthe infrared region. The light represents visual information about theposition and orientation of the mobile apparatus 21 that is transmittedto the plurality of cameras 1 via an optical lens 11 which in theillustrated embodiment has an infrared filter 9 mounted over the opticallens 11. The infrared filter 9, although not necessary, attenuates thenoise caused by other lights in a factory environment by limiting thelight that is applied to the cameras 1 to light that is in the infraredportion of the light spectrum.

The plurality of cameras 1 convert the visual information to a videosignal that is applied via a conductor 13 to the control and navigationcircuit 15 which receive operator inputs from a terminal 17 and convertsthe video signals to location commands and apply the location commandsthat contain information as to the mobile apparatus's 21 position andorientation to a communication system 19 which can be a radiotransmitter or in a factory environment more preferably light or aninfrared wireless communication system. The infrared wirelesscommunication system through the use of light emitting diodes and lightsensitive diodes is able to communicate between the control andnavigation circuit 15 and the mobile apparatus 21 on which the lightsources 3, 5 and 7 are mounted. The mobile apparatus 21 compares itsposition with its commanded position and will provide motion commands toits wheels or legs 23 for relocating its position and/or additionallymove a robot arm 25 to perform a commanded task. The position of the arm25 may be determined by an optional beacon 27 or by internal referenceswithin the mobile apparatus 21.

Control and Navigation Circuit 15

In FIG. 2, to which reference should now be made, there is shown asimplified block diagram of the control and navigation circuit 15. Theplurality of TV cameras 1A through 1N provide visual information to thecontrol and navigation circuit 15 and more particular to a VisionInterface Module (VIM) 31. The vision interface module 31 selects apredetermined TV camera 1S which is a member of the group of TV cameras1A through 1N and as such not specifically identified in FIG. 2. Thevision interface module 31 converts the visual information that is inthe form of an analog signal having a waveform that corresponds to lightintensity versus raster scan position of the selected TV camera 1S intoa digital signal that provides a gray scale digital representation ofthe analog waveform that is provided by the selected TV camera 1S.

The gray scale digital representation of the analog waveforms isprocessed by and stored in a Picture Memory Module (PMM) 35 as atwo-dimensional array so that the intensity of the light by its positionwithin the memory array corresponds to a position of the raster scan ofthe selected TV camera 1S. Additionally, bus 39 conveys the digitalsignal to a Visual Histogram Processor (VHP) 33 where the digital signalis converted to a signal (herein after called a spatial domain signal)that represents the spatial domain position of the light sources 3, 5and 7 and thus the mobile apparatus 21. From the spatial domain signalthe centroids of the more intense light that is received by the selectedTV camera 1S are identified.

A host CPU and memory 37 that includes a navigation CPU and memory 37a,a Communication CPU and memory 517, and a control CPU and memory 505 andin particular the navigation CPU and memory 37a compares the centroidsof the light sources to an expected pattern stored within the navigationCPU and memory 37a of the light sources 3, 5 and 7 that are mounted onthe mobile apparatus 21. Once the pattern has been identified, then theposition including location and orientation of the mobile apparatus 21is transmitted to the mobile apparatus 21 via the communication CPU andmemory 517 and an infrared wireless communication system 19.

The infrared wireless communication system 19 includes an array ofinfrared light emitting diodes 45 for transmitting light to an infraredlight receiving diode 47 and conveys information and commands to themobile apparatus 21 based upon its location. Commands are also providedfrom a factory computer 17 or a terminal to the host CPU and memory 37for transmitting to the mobile apparatus 21 via the communication CPUand memory 517 and infrared wireless communication system 19. Anoperator may monitor the movement of the mobile apparatus 21 via a TVmonitor 29 that is connected to the vision interface module 31. Theoperator terminal 17 may also be used to input commands directly to themobile apparatus 21 for purposes of testing or for manual control.

Vision Interface Module 31

FIG. 3 to which reference should now be made is a block diagram of thevision interface module 31 of FIG. 2. The video signals from theplurality of TV cameras 1A through 1N are brought into the visioninterface module 31 via conductor bundle 32. The video signals areanalog signals and are applied to an analog multiplexer 49 which basedupon an output from the navigation CPU and memory 37a via ControlRegister Unit (CRU) output lines 41 selects the predetermined TVcamera's 1S video signal for processing. The navigation CPU and memory37a selects the camera's 1S video signal based upon itspredetermnability that is established primarily by the criteria of thecamera that has the best view of the mobile apparatus 21 given its lastknown position and is expected direction of motion after the executionof the last move command. The video signal is multiplexed by the analogmultiplexer 49 and applied to a video clamp or active clamp 51 forclamping of the video signal to a reference voltage that is establishedby the clamp voltage potentiometer 42. The clamped video signal isapplied to amplifier 55 and then to a flash A to D converter 57 whichtakes a digital snap shot of the clamped and amplified video signal.

Reference voltages from D to A converters 53 and 59 depending on thecommands present on data bus 41 from the control register unit of thenavigation CPU and memory 37a compensate for input video signal rangevariations.

The vision interface module 31 also provides on conductor 40 a videosignal to the TV monitor 29 where an operator may monitor the progressof the mobile apparatus 21. This is handled by taking a frame of thedigital video information from the picture memory module 35 via data bus43 and applying it to a latch 61 for conversion to an analog signal by adigital to analog converter 63. The analog signal is applied to a videocomposer or mixer 65 for mixing with a video overlay signal to reproducea replica of the previously stored video signal. A buffer amplifier 67buffers the reproduced video signal prior to application to the TVmonitor 29.

The mixer 65 mixes the converted analog signal with external overlayinputs that are provided from the picture memory module 35 and thenavigation CPU and memory 37a to obtain an accurate representation ofthe video image. Additionally, provided in the vision interface module31 is a synch and timing generator 69 that provides the pixel clocks,the horizontal synch pulses and the vertical synch pulses for ensuringproper synchronization of the video signals.

An interface is provided by the processor interface 71 which interfacesthe output of the control register unit from the navigation CPU andmemory 37a to the vision interface module 31. The processor interface 71provides line receivers so that the control register unit signals arereceived properly before application to the appropriate modules withinthe vision interface module 31.

Visual Histogram processor 33

FIG. 4 is a block diagram of the visual histogram processor 33. Thedigitial video signal is applied from the vision interface module 31 viadata bus 39 to the visual histogram processor 33. The portion of thedigital video signal which carries synch and reference clock data isapplied to the window generator 73. The window generator 73 generates awindow around the images that are predicted to be the images of thelight sources or beacons 3, 5 and 7 to reduce the area of examinationfor identifying the objects, such as light sources or beacons 3, 5, and7. Once a window is generated the output of the window generator isapplied in combination with data from a video threshold circuit 920 to arow projection counter 4 that is used to perform a row vector summationfor each row with the summation stored by the fly control softwareexecuted by the CPU 8 and stored in memory 6. FIG. 8 provide a moredetail instruction of the functions implemented by the CPU 8. Similarlythe output of the window generator 73 is applied to a column addresscounter 2 that generates the memory addresses (herein after referred toas column address data) corresponding to the raster column positions.

The column address data is split into two parallel paths and applied toa first and second accumulator processing circuits reducing thereby theaccumulator processing rate to one half the input pixel rate. During theactive window area, memory locations corresponding to the position ofthe raster scan of the selected camera 1S are recalled and selectivelyincremented as a function of the processed video signal from the videothreshold circuit 920. The first accumulator processing circuit includesan accumulator 922, an adder 14 and a "D" type latch 16. The secondaccumulator processing circuit includes an accumulator 924 an adder 22on a "D" type latch 24. The first and second accumulator circuits workin tandem and process all columns data that occurs within the windowthat is generated by the window generator 73. Under command of the CPU8, D type latch 10 and D type latch 18 are held in a reset conditionduring this mode of operation (the accumulation of data within thegenerated window) and the carry in pins of adders 14 and 22 are held ata logic high by the action of inverters 12 and 22. When control line 921is held low by the action of CPU 8 the accumulator adds the gray scalevalue of each pixel in the corresponding columns.

Video histograms are performed by the action of CPU 8 on a multiplexer923 which directs the digital video signal into the accumulator addressbus. Through the same action as above the accumulator generates ahistogram in the accumulator memory.

The information contained within the accumulators 922 and 924 iscompiled within the CPU 8 and is stored within its memory 6 as well asconveyed to the navigation CPU and memory 37a via data bus 41.

Window Generator 73

The window generator 73 is illustrated in FIG. 5, to which referenceshould now be made, and includes a horizontal address counter 77 and avertical address counter 75. The horizontal address counter 77 and thevertical address counter 75 receive horizontal synch, vertical synch,and pixel clock from the synch and timing generator 69 that is containedwithin the vision interface module 31 and generates a count thatcorresponds to the scan position of the selected raster scan TV camera1S. The horizontal address counter 77 provides a horizontal address to ahorizontal window memory 88 which has a memory size that corresponds tothe number of horizontal addresses of the pixels of the members of theplurality of cameras 1. As the horizontal address counter 77 cyclesthrough its capacity it will address all of the memory locationscontained in the horizontal window memory 88 that correspond to thepixel location of the raster scan cameras 1. Similarly, the verticaladdress counter 75 addresses the vertical window memory 76 which memorysize corresponds to the vertical addresses of the pixels of the membersof the plurality of cameras 1. CPU 8 via data bus 925 sets up anapproximate location of each window that is to be generated by storingin the horizontal and vertical window memories 88 and 76 a logic stateat the start and stop addresses that will toggle J-K flip-flops 79 and91 to generate the windows at the desired locations. Additionally, usingthis technique allows for generating as many windows as needed, onlybeing limited by the memory sizes.

The windows are used to provide the vector summation of the videosignal. The window is generated according to the coordinates of theposition of the raster scan of the selected TV camera 1S as provided bythe CPU 8.

The data provided includes a start coordinate and an end coordinate andeach coordinate has a horizontal and vertical position. The horizontalstart position is encoded by CPU 8 setting a bit in the horizontalwindow memory 88 corresponding to the start position. The coordinate ofthe stop or end of the window is encoded by setting a bit at the addressjust previous to the address corresponding to the stop position inhorizontal window memory 88. This permits the inclusion of an additionalstart bit at a position corresponding to a window which is exactlyadjacent to the previous window. The output of the horizontal windowmemory 88 appears as two pulses, one occurring at scan positioncorresponding to the start and one at the stop of the window. Additionalstart stop pairs may be set to produce multiple window areas.

J-K flip-flop 79 is reset to the Off state by the action of thehorizontal synch. Corresponding to the screen coordinates, thestart/stop pulses toggle J-K flip-flop 79.

Index counter 81 produces a unique index for each window to directresults of each vector summation into a separate portion of accumulator922 and 924.

Similarly, the start coordinate for the vertical position is stored invertical window memory 76 and locations corresponding to the start andstop positions are encoded by setting a bit in the vertical windowmemory 76 corresponding to the start and stop positions. Compensationfor adjacent windows and vector sum memory index generation is provided.

A composite of the vertical window VWIND, and the horizontal window isproduced by the action of timing signals generated by horizontal windowmemory 88 and vertical window memory 76. J-K flip-flop 83 is set intothe On condition by the presence of the horizontal start pulse, verticalwindow and the absence of a previous window. The previous function isperformed by the action of inverter 87 and NAND gate 84. J-K flip-flop83 is reset on the condition of the presence of the horizontal andvertical window and the stop bit from the horizontal window memory 88.This action is delayed one clock period and is inhibited by the presenceof an adjacent window which as the effect of keeping flip-flop 83 in theset condition. The previous function is performed by the action offlip-flop 79 which registers the presence of a horizontal window periodand NAND gate 90. Flip-flop 89 performs the one clock delay. AND gate 85inhibits the action in the presence of an overriding adjacent windowrepresented by the presence of an additional start pulse.

The vertical window or VWIND signal is generated by the action ofvertical address counter 75 and vertical window memory 76 as describedabove. The start/stop pulses are applied to J-K flip-flop 91 by theaction of AND gate 92 which combines the vertical window start pulsewith the HCLR signal. HCLR is a one clock period wide reset pulseoccurring once per line and generated from the HORZ SYNC signaldescribed above and generated in the synch and timing generator 69. Thishas the effect of toggling J-K flip-flop 91 On. The output of J-Kflip-flop 91 is applied in combination with the vertical start signalVRAM and horizontal timing signal HCLR. Logic gates 93, 96 and 95control the action of J-K flip-flop 97 which defines the vertical windowperiod. The action of J-K flip-flop 98, D type latch 99 and NAND gate 94produce a last active line flag, which is used to resynchronize indexcounter logic 81.

Picture Memory Module 35

The video bus 39 transmits the video signal from the vision interfacemodule 31 to the picture memory module 35 which is controlled by a CPUand memory combination 46. The CPU and memory combination 46 providesCRU control to the logic that is contained within the picture memorymodule 35 and also provides interface to the navigation CPU and memory37a of FIG. 1.

Referring to FIG. 6, a multiplexer 30 will select either the output froma video picture buffer ram 40 or the input on the video bus 39 forprocessiing as instructed by a CPU and memory combination 46. Theselected digitized video signal is applied to a binary logic map 34which is a RAM that is coded to convert the information coded in thedigitized video signal from gray scale to black and white codedinformation. The function performed by the binary logic map is usually athreshold comparison. The comparison is such that when the lightintensity level that is coded in the digitized video signal is above thethreshold a first logic state is obtained and when below the threshold asecond logic state is obtained as an output from the binary logic map34.

The black and white digitized video signal that is provided on theoutput of the binary logic map 34 is applied to a 3×3 processor 36 whichcompares a 3×3 matrix of logic states. This 3×3 processor uses a digitalfiltering technique which eliminates noise by comparing a 3×3 matrix ofpixel information coded in the digitized video signal, and if the centerpixel is one state and its adjacent members are different states, thenthe state of the center pixel is changed to be in agreement with theadjacent state of the surrounding pixels. The output of the 3×3processor 36 is applied to a video picture buffer 1140 which is a randomaccess memory and is stored under control of the CPU and memorycombination 46 and an address counter control logic 44.

If selected for display via the TV monitor 29 through a vision interfacemodule 31 a multiplexer 1142 enables the display to be the output of thevideo picture buffer 1140, the processed data from the 3×3 processor 36or alternately the video data that is provided on the video bus 39. Ablanking delay 26 compensates for processing delays incurred within thepicture memory module 35.

Theory of Operation

The theory of operation may be understood by referring to FIGS. 7 incombination with FIGS. 1 through 6. FIG. 7A is a grahic representationof the analog signal from the selected camera 1S that after beingmultiplexed by the analog multiplexer 49 of FIG. 3 is applied to theactive clamp 51. As was discussed earlier in the specification, the TVcameras 1 are raster scan cameras in which scans are made over the fieldof view to obtain a picture 100 that is comprised of a plurality ofscans 101. In the case where light intensity is different than thebackground then an image 103 is depicted in the picture 100. Multipleimages as in scan 105 are additionally depicted. Each scan as it occursis multiplexed and applied to the active clamp 51 the output of which isillustrated in FIG. 7B in which the scan 105 has two blips 107 and 109that represent light intensity.

The output of the active clamp 51 is illustrated in FIG. 7C in which atime discrete digitization operation is illustrated by waveform 111.Waveform 111 is an illustration of the time discrete digitization ofwaveform 105 which is the input signal to the active clamp 51. Eachdiscrete segment in the illustrated embodiment is referenced to avoltage that is provided by the clamp voltage potentiometer 42. The timediscrete digitization operation is in synch with the pixel clock. Theoutput of the active clamp 51 is applied via buffer 55 to the flash A toD converter 57 which at the occurrence of every pixel clock converts thesampled video signal to a digital signal an example of which isillustrated by waveform 130 of FIG. 7C. The time discrete digitizationvoltages provided by the D to A converters 53 and 59 are used to set thebackground of the image that is being quantized.

In FIG. 7D, the synch and timing generator 69 output is illustrated andincludes a field index signal 113 and is generated by a request from thenavigation CPU and memory 37a via the control register unit to the synchand timing generator to indicate or to strobe a particular frame ofinterest for the navigation CPU and memory 37a. Additionally, the synchand timing generator 69 provides a vertical synch pulse that isrepresented by waveform 115 which is the vertical synch for a frame, anda horizontal synch signal is provided and represented by waveform 117which shows the horizontal synch pulses that are necessary to develop aframe such as that shown in FIG. 7A.

FIG. 7D also illustrates the number of conductors on the data bus 39 ofFIG. 2. Central to the group of circuits that includes the visioninterface module 31, the visual historgram processor 33 and the picturememory module 35 is the concept of a generalized video bus, digital bus39. The most important feature of the video bus, 39, is there is not arequirement for a pixel address bus. The vertical synch as representedby waveform ll5 and horizontal synch as represented by waveform 117,synchronizes each of the three modules. A field index waveform 113 andthe window waveform, 119, which can be also used as a pixel validaddress is also illustrated. With this configuration, the video bus 39can be reduced in size and handle a large variety of possible imagers,video cameras, without hardware changes. With this reduced number ofpins, modules can be configurated easily to select multiple digitalinputs. The window generator 73 of FIG. 4 provides windows as areillustrated by waveform 19 and in particular by output pulses 121 and123. The pixel clock is provided by the synch and timing generator 69and is represented by waveform 125 and the video data is represented bywaveform 127 with the output of the flash A to D converter 57 beingillustrated by diagram 129.

As the selected frame of reference is being digitized, the area withinthe window that is generated by the window generator 73 is processed bythe visual histogram processor 33. The resulting image (diagram 131 ofFIG. 7E) is also stored in the video picture buffer 1140. The imagesthat represent the beacons, because of the processing, are identified asperturbations such as at location 134. A ghost image which can beattributed to reflection or to other phenomena such as quantizationerror is illustrated at 136.

Horizontal and verical vector summations are performed on the digitaldata that is shown at diagram 130 to obtain a horizontal vectorsummation that is represented by waveform 133 and a vertical vectorsummation that is represented by waveform 135. The vector summation isperformed with the accumulator processing circuits that are a part ofthe visual histogram processor 33 illustrated in FIG. 4 and discussedtherewith.

The vector summations are thresholded by the operation of the CPU 8 andmemory 6 and projected by the operation of the circuits within thepicture memory module 35. The resulting waveforms are illustrated inFIG. 7F where waveform 137 illustrates a normalized projection of thewaveform 133 and waveform 139 illustrates the normalized projection ofthe waveform 135. The waveforms are combined and areas of coincidencesuch as at area 141 and 143 or the shaded areas represent areas of highprobability of containing a beacon image. Based upon the intensities ofthe areas of each image within a candidate box, a decision can be madeas to the exact position of the beacons of the mobile apparatus 21. Thusfrom the information illustrated in FIG. 7E, the location of the mobileapparatus 21 is achieved.

FIG. 8 is a flow diagram of the sequences that are programmed in thenavigation CPU and memory 37a, the CPU 8 and memory 6 of the visualhistogram processor 33 and the CPU and memory combination 46 ascontained within the picture memory module 35.

Beacon Centroidification and Transformation

Upon initialization, (FIG. 8A) the control and navigation system 2A isin a starting state 200. A prediction is made to select one of thecameras 1A-1N for scanning at block 201. After selection of the selectedcamera 1S the picture is applied via a video signal from the selected TVcamera 1S to the vision interface module 31 at block 203. At block 205,the window information is provided to the window generator 73 ofcandidate object positions from the CPU 8. At block 207 windows arebuilt around the coordinates of the suspected location of the mobileapparatus 21. The picture memory module 35 is used to discard falsecandidates and objects which are too large and to calculate thecentroids of the candidates which were illustrated in FIG. 7E at areas134 and 136 and store into the CPU and memory 37a at block 209. Thecentroid coordinates are transformed into a beacon plate/floor space atblock 210.

Beacon Number and Chord Length Acquisition

After the transformation of the centroid coordinates onto the floorspace, a count of the objects' centroids is made to determine how manyobjects are visible. This is performed at decision block 211 of FIG. 8Bvia tie point P2. The image is discarded at block 212 if there is only 0or 1 centroid identified and returns to start point 200 via tie pointP1. If there are two images visible, then this may indicate that thereis a presence of a mobile apparatus 21 and one of the beacons, either 3,5 or 7 is blocked by an object such as the robot arm 25 and the thirdimage needs to be synthesized. This occurs at block 213 and will bediscussed in conjunction with FIGS. 14 through 18. If four or moreimages are present, then a decision is made as to whether or not thereis a reflected image, and if there is a reflected image, then thereflected image is discarded and only the primary image of the mobileapparatus 21 is used. This occurs at block 215 and will be discussedlater in conjunction with FIGS. 11, 12, and 13.

In the situations where as is expected, three of the images are visible,the hidden image coordinate has been obtained or the reflected imagefiltered out then the three chord lengths between the images areobtained at block 215. At decision block 217, a decision is made toensure that the lengths of the chords are in proper proportions. If thelengths are not in proper proportions, then the control and navigationsystem 2a returns to tie point P1 and start position 200 to repeat theprocess. If the chord lengths are in proper proportions then a clockwiseordering of the chords from the hypotenuse is obtained at block 219. Thechords are then arranged in order to decreasing length at block 221 orif this arrangement can not be made at block 221, then the control andnavigation system 2a returns via tie point P1 to starting point 200.

Tie point P3 connects FIG. 8C with 8B and reference should now be madeto FIG. 8C.

Acquiring Mobile Apparatus Position and Heading

At block 223, the angle of the chords is computed and a decision is madeat decision block 225 to ascertain if one of the chord angles is 90degrees, or any other predefined degree angle. If the chord angles arecorrect, the perimeter or sum of the chords is computed at block 227. Ifnot, then through tie point P1, the control and navigation system 2areturns to start point 200. If the perimeter is the correct length, thenthe floor coordinate of the midpoint of the beacon hypotenuse iscomputed at block 229. If the perimeter is not the expected length, thenvia tie point P1, the control and navigation system 2a returns to thestart position 200.

The mobile apparatus' 21 heading is computer from bow and port chordangles at block 231. The bow chord is rotated 90 degrees and the mean ofthe port chord and the rotated bow chord is used as the final mobileapparatus heading. Communication of the heading and positional fix tothe mobile apparatus 21 is performed at block 232.

To ensure heading and positioning of the mobile apparatus 21 it isobvious from the discussion of the flow diagram that at least one anglehas to be at a particular degree that is different from the other twoangles. In the case of the embodiment of FIG. 1, the angle is 90 degreesand given the 90 degrees angle then the heading of the mobile apparatus21 and its relationship to the position of a raster belonging to anidentifiable camera that is a member of the group of cameras 1A through1N and consequently its position on a surface can be obtained. Thus, theprocess is repeated via connect point P4 to obtain a sequence ofpositions and the mobile apparatus 21 is guided across a factory flooror warehouse floor or any other location of its assigned duties throughits internal guidance system and the sequential updating of itsposition.

It should be noted that the angular relationship of the beacons to oneanother should provide an image of an asymmetrical triangle and theoptimum triangle is a 30, 60 and 90 degree triangle.

FIG. 9 illustrates the beacon identification and transformation stepsvisually by showing a plurality of cameras 1A and 1B. Each camera has afield of view that is projected at a height that is illustrated bydimension lines 151 and 153. The field of view covers a plane that isillustrated by the plane 155 for camera 1A and for camera 1B the plane157. These planes are calculated to be at the height of the beacons 159and 161. However, the height of the beacon plane is different than thefloor 163. Therefore, after the beacon plane at 155 or 157 iscalculated, the plane is then transformed to the floor space (block 210of FIG. 8A) to coincide with the plane 163.

Reflected Image Filtering

In the discussion of FIGS. 8 and in particular with FIG. 8B, at step orblock 214 that is implemented by the control and navigation system 2athat was disclosed in FIG. 1 the step of discarding of reflected imageswhen there are more than the expected number of images received andprocessed was identified. This step is necessitated by the operation ofthe mobile apparatus 21 in environments such as that normallyencountered in a manufacturing facility. A reflecting surface such as awindow 400 (FIG. 10) reflects the image of the mobile apparatus 21 tothe camera 1 so that the camera 1 visualizes an image that isillustrated in FIG. 11, i.e. two mobile apparatuses 21 and 21R. Thecontrol and navigation circuit 15 determine which image is the primaryimage and which image is the reflected image and discards theinformation that pertains to the reflected image. To implement thisreflected image filtering technique in the embodiment shown in FIG. 10,the beacons 3, 5 and 7 are arranged in an asymmetrical triangleconfiguration having as indicated by dimension lines 401 a 90 degreesangle between a line that is drawn from beacon 3 to 5 as compared to aline that is drawn from 5 to 7. The three beacons make a right handedasymmetrical 90 degrees triangle. The reflected image is an asymmetricalleft handed 90 degrees triangle and by knowing this information, thecontrol and navigation circuit 15 will disregard the reflected image21R.

The process that is utilized is illustrated in FIG. 12 and begins at astart point at block 214 which is also the reference point from FIG. 8B.All of the centroids of the images are obtained at block 402. Thedistance from each image centroid to a plumbed center point coordinateis obtained at block 403, and at block 405 a sort is made of the plumbpoint to centroid chords in order of lengths. At block 407, the imagescorresponding to the shortest center line chords which form a triangleof correct shape and size are identified and the coordinates of thethree selected beacons are obtained at block 409. At the end block 412the microprocessor returns to the execution of the next step whichoccurs at step 214 of FIG. 8B.

Hidden Beacon Identification Procedure

In FIG. 13 there is illustrated a manufacturing arrangement wherein twomobile apparatuses 21A and 21B are operating under the field of view ofa selected TV camera 1S. Each mobile apparatus 21 be it 21A or 21B hasthree navigational beacons 3, 5, and 7 and a robot arm 25. In both casesillustrated in FIGS. 13 and 14 one of the three beacons per mobileapparatus is obstructed from the field of view of the selected videocamera 1S. The condition occurs when the view of one of the beacons tothe selected camera 1S is obstructed and the control and navigationcircuit 15 will implement at block 213 of FIG. 8B the hidden beaconprocedure.

The problems that are solved by the hidden beacon procedure areillustrated in FIGS. 14 through 17 which should be used in conjunctionwith FIG. 13. The view of the beacons as seen by the selected TV camera1S is shown by the field of view 416. FIG. 17 is a detailed view inwhich two beacons are visible and a third beacon is blocked. In bothFIGS. 14 and 17, the plumb point is marked "P". Image 414 shows how theblocking of beacon 3A raises two distinct possibilities: it can beeither where beacon 3A is illustrated or where beacon 3AR isillustrated. Image 415 illustrates when one of the other two beacons isobstructed from view of the selected TV camera 1S. In the case of image415 the robot arm 25B obstructs beacon 5B. The control and navigationsystem 2a must decide the exact location of the beacon and the choicesare at the location of beacon 5B or the location of beacon 5BR.

The solution to the above discussed problem may be understood byreferring to the geometry of the image that is viewed by the selected TVcamera 1S. This is illustrated in FIG. 15 where there is a triangle withthree sides s1, s2, and s3 and where side s1 is the chord betweenbeacons 3 and 5, side s2 is the chord that connects beacons 3 and 7 andside s3 connects beacons 5 and 7. The triangle is in a plane 417 thatcontains beacons 3, 5, and 7.

Procedures for Identifying Hidden Images

FIGS. 16 are flow diagrams of the procedures for identifying hiddenimages such as those shown in FIG. 13. If as shown in FIG. 17 the imagesof the two beacons are identified as "A" and "B", the control andnavigation circuit 15 is initiated at block 420. The coordinates ofimages A and B are supplied to the control and navigation circuit 15 atblock 421. The distance between images A and B is obtained at block 423.The control and navigation circuit 15 has stored therein the lengths ofthe chords s1, s2, and s3. At block 424 the control and navigationcircuit 15 compares the distance between images A and B with the lengthof each chord s1, s2 and s3 and selects the chord which has a lengththat closely matches the length of the distance between images A and Band assigns the selected chord the reference notation "D" as isillustrated in FIG. 17. As is indicated in FIG. 16a at block 425, theunselected chords are designated by the reference letters "E" and "F". Afirst circle AE of a radius E is constructed around A and a secondcircle BE of equal radius is constructed around B. A third circle AFhaving a radius length F is constructed around A and a fourth BF ofequal radius around B.

The next step is to obtain the coordinates of candidate points C1 and C2at the intersections of circles AE and BF. This step is implemented atblock 427. A clockwise ordering of the chords AB, AC1 and BC1 isperformed at block 431. A decision is made at block 433 as to whetherthe length of the chords is increasing in length or decreasing in lengthstarting with the hypotenuse. If the clockwise arrangement of the chordsproduces an arrangement which begins with the hypotenuse and decreasesin length then the control and navigation system 2a goes to block 439and coordinate C1 becomes the first primary candidate and is designatedas X1. Otherwise, the control and navigation system 2a goes to block 437where C2 is designated as the first primary candidate X1.

At block 44l of FIG. 16B, the coordinates of the points of intersectionof circles AF and BE are obtained and designated C3 and C4 respectively.The lengths of the chords (X1, C3) and (X1, C4) are obtained at block443. A decision is made at block 445 to ascertain if the chord (X1, C3)is shorter than (X1, C4). If (X1, C3) is shorter than (X1, (C4) then C4is selected as the second primary candidate X2 at block 447; if not, C3is selected as the second primary candidate X2 at block 449.

The distance is computed between the coordinates of the first and secondprimary candidates C1 and X2 and plumb point P at block 451. If thelength of the chord PX1 is less than PX2 which decision is made at block453 then X2 is the third beacon point. Otherwise, X1 is the third beaconpoint as indicated at block 457. The results of the identification areprovided at block 459 to complete the hidden beacon procedure and thecontrol and navigation system 2A returns to block 215 of FIG. 8B.

The Mobile Apparatus 21

FIG. 18 is a block diagram of the mobile apparatus 21 according to theinvention as illustrated in FIG. 1. A radio or other wirelesscommunication system 19 such as infrared diodes receives position dataand commands from the navigation CPU and memory 37a and transfers thatto the heart of the mobile apparatus 21, a CPU and memory 52. The CPUand memory 52 transfers instructions to a robot art or material handlingapparatus such as that illustrated in FIG. 1 as a robot arm 25 and alsoprovides commands to the navigation beacons such as "turn on", "turnoff", "flash" or other functions. Additionally, the CPU and memory 52provides steering, start, stop and rotation instructions to a pluralityof wheel assemblies 58. Each wheel assembly includes a first D to Aconverter 60, a steering servo amplifier 62, a steering motor 64 whichis responsive to commands from the steering servo amplifier 62, a secondD to A converter 72, a drive servo amplifier 70 and a drive motor 68which causes the wheel 23 to rotate. Both the drive servo amplifier 70and the steering servo amplifier 62 are illustrated as servo amplifier371 in FIG. 19. Both the first D to A converter 60 and second D to Aconverter 72 are illustrated as D to A converter 372 in FIG. 19. Bothsteering motor 64 and drive motor 68 are illustrated as motor 376 inFIG. 19. A battery 48 provides power to the mobile apparatus assembly21.

CPU/Memory Combination's 52 Program Sequence

The sequence of events that is executed within the CPU and memory 52 ofthe mobile apparatus 21 begins when the mobile apparatus 21 via theradio or wireless communication system 19 receives a go to goal commandand verification of the receipt of that command is provided. Thenavigation system initially provides the positioning information to themobile apparatus 21 and a comparison is made to ascertain if the currentposition is equal to the goal position. If it is, then the CPU andmemory 52 will cause the mobile apparatus 21 to stop and to wait until anew go to goal command is received. If it is not the same, then thedirections, and displacements to the goal position are ascertained andan update of the steering angles and velocity towards the goal positionare implemented that will steer the mobile apparatus toward the goalposition.

Mobile Apparatus Guidance and Servo Control System

In FIG. 19 there is shown a simplified circuit diagram of a servocontrol loop 320 that is contained within the mobile apparatus 21 foreach wheel assembly 58 and is used to control the mobile apparatus 21.The servo control loop 320 includes the CPU and memory 52. The CPU andmemory 52 provides in the form of digital signals speed and directiondata to the servo amplifier 371 via the D to A converter 372. The D to Aconverter 372 converts the digital signals that are provided from theCPU and memory 52 to an analog voltage or command voltage that isapplied to the servo amplifier 371. The servo amplifier 371 providesbuffering to the D to A converter 372 and amplifies the command voltagewhich is applied to a motor 376 and is used to cause the motor 376 torotate a shaft 374 which causes the load 23, such as a wheel or steeringwheel, to rotate the number of degrees as represented by the amplifiedcommand voltage that is applied to a motor 376 from the servo amplifier371. A shaft encoder 333 encodes the degrees of rotations and providesthis information via conductor bundle 375 to an encoder interface 370which converts the output of the shaft encoder to signals that areacceptable by the CPU and memory 52.

Dead Reckoning Guidance System

The mobile apparatus 21 is permitted to be free roaming and as such thesteering angles and drive velocities of each wheel must be coordinated.Referring to FIG. 20 which should be used in conjunction with FIGS. 18through 19 there is shown the implementation of a dead reckoningguidance system for the mobile apparatus 21. The CPU and memory 52receives a commanded trajectory from the control and navigation circuits15 via the wireless communication system 19 and implements a positioncomparison at block 331. The results of the comparison are used toformulate a wheel position error that is used by the CPU and memory 52for developing of trajectory control and wheel coordination commands atblock 328 for application to the wheel assemblies 58 in the form ofdigital signals that was described in conjunction with FIG. 19. Thedigital signals are applied to the servo control at block 327 whichgenerates commands for the servo amp and mechanism 323 which includesthe D to A converter 372, a servo amplifier 371 and the motor 376. Afterthe conversion to analog signals by the D to A converter 372, the analogcommand voltage is applied to the servo amplifier 371 for amplification.The servo control 327 receives an angle via an angle sensor 325 and usesthis information to adjust the amplitude of the command voltage that isapplied to the servo amplifier and mechanism 323. Additionally, thesensed angle as measured by the angle sensor 325 on each motor 376 isapplied to a position estimator 329. The angle sensor 325 includes shaftencoder 333, conductor bundle 375 and encoder interface 370. Theposition estimator 329 also receives a total of 6 position datas from 6angle sensors 325 and is directed to a 3 wheel mobile apparatus 21 witheach wheel assembly 58 being used for both drive and steering functions.The position estimator provides the position of the mobile apparatus 21to the position comparator 331 which generates commands to adjust theposition of the mobile apparatus 21 until the commanded position isachieved. Periodically the mobile apparatus 21 receives measuredposition from the control and navigation circuit 15 and corrects thedead reckoning position accordingly. The equation of motion for eachwheel is provided by equation 1; equation 2 provides the wheelcoordination and equation 3 provides the dead reckoning determination.All of the equations are found in the table of equations.

FIGS. 21 are a flow diagram of the software implementation for guidanceof the mobile apparatus 21 of FIG. 1 in which the control and navigationcircuit 15 transmits to the mobile apparatus 21 command instructions tocommand the mobile apparatus 21 to traverse a predetermined commandedpath. Additionally, the control and navigation circuit 15 periodicallyprovides to the mobile apparatus 21 position data which the mobileapparatus 21 uses to correct for errors that are present in navigationcommands that are contained within a dead reckoning navigation procedurethat is executed by the mobile apparatus 21.

The CPU and memory 52 executes within the CPU section the dead reckoningprogram that is stored within the memory of the CPU and memory 52 isrepresented by the flow diagram contained within FIG. 21. At block 350the control and navigation system 2A formulates the absolute position ofthe mobile apparatus 21 and transmits it via the radio or other wirelesscommunication system 19 to the mobile apparatus 21. The mobile apparatusas indicated at block 351 is at a stop position where it waits for theabsolute position data to be provided from the control and navigationsystem 2A which is used in the embodiment of FIG. 1 as a sensor ratherthan a controller. At block 353 the vehicle, mobile apparatus 21, beginsmotion in a commanded direction that is provided from the control andnavigation circuit 15 and stores the extrapolated positions and vehiclereference time stamped after each time period. At block 355 the vehicle,mobile apparatus 21, receives absolute position data with sensorreference time stamp from the control and navigation circuit 15, and theCPU and memory 52 searches stored extrapolated data for the closestmatch between the absolute data position and the extrapolated radialmove from the stop position at block 351 to the current position atblock 357.

The CPU and memory 52 then determines the offset between the time stampof the closest match position from block 357 and the absolute positionsensor data at block 359. The mobile apparatus 21 proceeds in thecommanded direction continually storing the extrapolated position andgenerating the reference time stamp after each time period at block 361.

Once the mobile apparatus 21 receives absolute position data with timestamp from the control and navigation circuit 15 then a comparison ismade to see of a preselected time limit between data points is exceeded.If so, then line 363 is taken from block 365, and a failuresynchronization in timing between the control and navigation circuit 15and the mobile apparatus 21 occurs at block 367. Therefore, the mobileapparatus 21 must stop and wait and return to block 350 until itreceives its absolute position. If the time limit between the datapoints is not exceeded, then the CPU and memory 52 proceeds to block 369where the CPU and memory 52 determines extrapolated position at the timewhen the absolute position was measured using known offset from block359 and the stored extrapolated position in the memory portion of theCPU and memory 52. At block 371 errors in the extrapolated position whenabsolute position was measured are subtracted from the currentextrapolated position.

If the errors are greater than expected, such as 6 inches in radialdistance or 5 degrees, then the assmuption at block 367 that there is asynchronization time failure is made, and the CPU and memory 52 commandsthe mobile apparatus 21 to stop. If not, line 373 is taken and thecontrollable mobile apparatus 21 proceeds in the commanded direction.

In the discussion of the hidden image subroutine that was discussed inconjunction with FIGS. 13 through 17, it was shown that when one of theimages is blocked that it is possible to know the position and headingof the mobile apparatus 21. However, in reflective environments or othercombination of environmental vision conditions, the hidden imagesubroutine may be inaccurate. Therefore, the mobile apparatus willoverride an erroneous command that is provided it from the control andnavigation circuit 15 of FIG. 1. This override ability is illustrated inFIG. 21C to which reference should now be made.

Within the execution of blocks 351, 355 and 365 of FIGS. 21A and 21B,the CPU and memory 52 leaves the start block 352 and waits for positiondata from the control and navigation circuit 15. Contained in theposition data is an indication of when there is a hidden image. Thisdata is checked at block 356. If there is no hiddem image indication,the program is exited via block 364. If there is a hidden imageindication, then blocks 358 and 360 compare the angle from the controland navigation circuit 15 with the dead reckoning angle. The angle asdetermined by the dead reckoning guidance will be the dominate angle ifthe two angles are 180 degrees different indicating an error in thehidden image routine data, and at block 362 the data from the controland navigation circuit 15 is adjusted based on light geometry andinformation comtained in the position data indicating which image washidden and thus incorrectly synthesized. This route of FIG. 21C isexecuted whenever position data is received from the control andnavigation circuit 15.

Velocity Filter for a Servo System

The CPU and memory 52 of FIG. 19 executes a servo filter program that isillustrated by the flow diagram in FIG. 22 to which reference should nowbe made. In FIG. 22 at block 376 the CPU and memory 52 calls thevelocity filter into play and the motor position or shaft position thatis determined by the shaft encoder 333 is read at block 377. Theunfiltered velocity is determined at block 374 based upon the degree ofrotation of the shaft encoder between reading times and the time betweenreadings. The CPU and memory 52 as does most microprocessors, operateson a cycle time basis so these periodic readings can be performed everyn cycles where n is a positive number that can be from 1 to infinity. Atblock 375, the filtered velocity is obtained by summing the previouslyfiltered velocity times a constant K1 with the unfiltered velocity timesa constant K2. The sum will equal the filtered velocity.

K1 and K2 have the limitation of the summation of which must be equal to1 and are determined by experimentally adjusting the value of the ratioof K1/K2 based on the dynamic performance of the servo system especiallyat low or slow velocities when the data from the shaft encoder 333 ischanging at a very slow rate. For example in the embodiment of FIG. 19it is possible for the shaft to be rotating at one quarter of anincrement per sample period that is performed by the CPU and memory 52.The velocity without filtering is 0 degrees per sample, for threesamples and 1 degree per sample, for the fourth sample. This of coursewill cause the servo system to lunge every fourth sample and thuspossibly damage any delicate material being handled by the mobileapparatus 21. The selecting of the constant K1 and K2 and utilization ofthe velocity filter will eliminate the problem by smoothing out thevelocity over all four samples.

Before returning to FIG. 22 at block 376, it should be noted that theinput and output of digital servo systems is quantized. In the exampleof FIG. 19, the incremental shaft encoder 333 has a quantization of +-one count. There are thus a maximum and a minimum value allowed forvelocity. This information is determined by the resolution of the shaftencoder. Thus at block 376 the filtered velocity is bounded by theminimum and maximum velocity based on the quantization of the shaftencoder 333.

Procedure for Selecting of the Constants K1 and K2

Set the ratio of K1 to K2 equal to a constant such as 1 and increase thevalue if the servo system twitches or decrease the value of the ratio ifthe servo system is unstable and oscillatory.

Apparatus Braking System

In FIG. 23 to which reference should be made there is shown a simplifieddiagram of a steering servo control system incorporating the embodimentsof the invention. Power is provided from a power source 381 to a powercontroller which controls the power that is applied to the servoamplifier 62 and the steering motor 64. A flying capacitor circuit 389applies the power from the servo amplifier 62 to the steering motor 64via relay contacts 407 to 405 and 411 to 413. Additionally, power isapplied to a capacitor 391, which is the heart of the circuit, from thepower source via relay contacts 393 and 403 and relay contact 401 to397. In the event of loss of power, then coil 385 will becomede-energized causing contact 411 to engage contact 409 and contact 405to engage contact 399 and similarly causing contact 397 to engagecontact 415 and contact 393 to engage contact 395, applying thereby, thecharge that is stored in capacitor 391 to the steering motor 64uncoordinating the steering of the mobile apparatus 21. Additionally, abraking command may be applied to coil 387 which will cause coil 385 tobe de-energized, once again applying the voltage stored in capacitor 391to steering motor 64. The application of the charge on capacitor 391 tothe steering motor 64 as discussed above is illustrated in FIG. 24.

FIG. 25 shows a four wheel embodiment of the mobile apparatus 21 inwhich wheels 23A and B are used to steer the mobile apparatus 21 byvarying the angle 417 and 418 that the wheels make with the mobileapparatus 21. Notice that in the steering mode that the angle that isrepresented by dimension line 417 is approximately equal to the anglethat is represented by dimension lines 418. In the uncoordinated mode ofoperation, or braking operations then the wheels 23A and 23B are drivenin the direction to oppose the motion of the mobile apparatus 21 andconsequently the angle that is represented by dimension lines 419 isuncoordinated and unrelated to the angle that is represented bydimension lines 420. Ideally, the angles should be equal, but inopposite direction of the center lines 425 and 426 of the axis of thewheels 23A and 23C, and 23B and 23D.

FIG. 26 is another embodiment of the invention in which there are onlythree wheels with the steering being implemented by wheels 23C and 23D.Dimension lines 421 and 422 represent the coordinated steering of thecontrollable mobile apparatus 21 and in FIG. 26B there is representedthe braking operation in which the angles that are represented bydimension line 423 and 424 are such to oppose the forward motion of thecontrollable mobile apparatus 21.

A Wireless Communication System

In FIG. 27 to which reference should now be made, there is shown a blockdiagram of the communication and navigation systems according to theinvention. As in the case of FIG. 1, a terminal 17 is provided so thatan operator may communicate with the control and navigation circuit 15that include as discussed earlier, the command circuits necessary togenerate command signals for communicating of commands to the mobileapparatus 21. The command signals communicate via a fixed command pod19A which transmits through blinking lights to the mobile apparatus 21and in particular to the mobile combined communication and navigationpod 19B which is contained within each beacon 3, 5 and 7 of FIG. 1.

The mobile combined communication and navigation pod 19B converts theblinking lights into digital data that is applied to a modem controller19C which demodulates the data for application to the CPU and memory 52.The navigation circuits then work as was previously discussed inconjunction with FIGS. 18 through 22. In the embodiment of FIG. 27, datais additionally modulated by the modem controller 19C for communicatingto the operator via the mobile combined communication and navigation pod19D and a fixed communication pod 19A where the control and navigationcircuit 15 includes a modem controller not shown, however it isidentical to that illustrated at 19C for demodulating of the data.

FIG. 28 to which reference should now be made, is a waveform diagram ofthe transmission between the mobile apparatus 21 and the control andnavigation circuit 15 via the fixed communication pod 19A which in theembodiment shown is mounted next to the TV camera 1. The navigationbeacon controls are designed to blink as is illustrated in waveform 433at blink points 434 and 435. The data from the modem controller 19C isillustrated by waveform 440 where FSK modulated data is provided inburst at points 436, 437 and 438. The actual transmission of data isillustrated by waveform 439 and can be visually perceived as waveform440 being combined with waveform 433 to obtain waveform 439 thatincludes the FSK modulated data at areas 442, 443 and 445 which istransmitted via the mobile combined communication pod 19B. Additionally,waveform 441 provides in the form of an illustration the reception ofdata as it is transmitted to the mobile apparatus 21 from the fixedcommunication pod 19A in which the navigation beacon controls asillustrated by waveform 433 are not required and thus not a part of thedata that is transmitted from the base station.

Navigation Pods

FIG. 29 is a top view of the mobile combined communication andnavigation pod 19B which are mounted on top of each of the beacons 3, 5and 7 of FIG. 1. A plurality of light emitting diodes 447 are arrangedin a parabolic shape for providing a concentration of light which in theembodiment shown is in the infrared region for transmittingcommunications and position information to a receiving station.Additionally mounted on the rim of the communication and navigation pod19B is a light receiving diode 449.

FIG. 30 is a sectional view of the communication and navigation pod 19Bas seen from section lines 31 and illustrates the parabolic shape 450 ofthe arrangement of the light emitting diodes 447 that are used totransmit both the data and beacon positioning from the mobile apparatusand just the data from the base station as illustrated in waveforms 439and 441 respectively.

Predicting Tracking Cameras For Free Roaming Robots

FIG. 31 illustrates a plan view of a factory floor or manufacturingenvironment in which four camera zones, a first camera zone 451, asecond camera zone 453, a third camera zone 452 and a fourth camera zone454, are illustrated which represents the viewing cameras of selectedmembers of the plurality of cameras 1 of FIG. 1. Dashed line 461represents a possible path for the mobile apparatus 21 which begins inthe first camera zone 451 and ends in the third camera zone 452. As longas the mobile apparatus 21 remains in zone 1, navigation contact ifmaintained by the overhead camera whose viewing angle coincides with thefirst camera zone 451. For some period of time after the mobileapparatus leaves the viewing angle or the first camera zone 451, themobile apparatus 21 will not be visible to any camera. This blind zoneis illustrated as area 463. Sometime later, the mobile apparatus 21 inthe embodiment of FIG. 31 will enter the third camera zone 452 and anormal navigation control will be established as was discussed inconjunction with in FIGS. 18 through 22.

However, for the period of time when navigation contact is lost in blindarea 463, there is uncertainty about the precise location of the mobileapparatus 21 and about the direction in which it is heading. In FIG. 31the mobile apparatus 21 makes a sharp turn as indicated by dotted line461 but in may actually make a number of turns during this time asillustrated by dashed line 456 in FIG. 32.

As the mobile apparatus 21 navigates its position and heading in thevisible zones 451, 452, 453, or 454, that information is acquired andstored within the memory portion of the navigation CPU and memory 37a ofFIG. 2. When the controllable mobile apparatus moves out of the zone ofvisibility, a radius of uncertainty is established around the storedposition. This radius is proportional to and at least equal to thedistance that the mobile apparatus 21 can achieve travelling at itsmaximum velocity. The area enclosed by the circles drawn around thisradius of uncertainty such as illustrated at 455 is compared with thezone of visibility of each of the effected cameras whose zone ofvisibility corresponds to the third camera zone 452 and the secondcamera zone 453. If there is an overlap between the two areas, then itis judged possible that the mobile 21 apparatus has entered the camerazone such as at area 465. If this is the case, then the vision interfacemodule 31 selects the input from that camera and analyzes the visualinformation in search for the image of the mobile apparatus 21.

In the situation where there is no overlap between the camera zones ofvisibility such as at area 463 then the region of uncertainty surroundsthe last known mobile apparatus's 21 position. The uncertainty radius isperiodically incremented over time to keep it accurately representativeof the mobile apparatus 21 escape distance until the radius has grown toequal the maximum diameter of the factory. The efficiency of thescanning is reduced to a minimum since no cameras will then be excludedfrom consideration. However, for short lapses of visibility which makesup the vast majority of the case in a typical factory the precedureillustrated in FIG. 33 results in a significant savings in time.

The navigation CPU and memory 37a in implementing the illustration ofFIGS. 31 and 32 utilizes the procedures shown by the flow diagramillustrated in FIG. 33 in which at the start position 457 the positionalgorithm is implemented and at decision block 458, the mobileapparatus's 21 position is determined to be previously defined or not.If it is not, then tie point P3 is taken. If it has been, then the firstcamera is selected by setting "n" the camera number, equal to 1 at block467. A decision is then made at block 469 to ascertain if the selectedcamera's zone of view overlaps the blind area or "region ofuncertainty". If it does not then a jump is made to block 475, However,if the selected camera's zone of view does overlap than the positionsubroutine proceeds to block 471. The selected camera zone is searchedfor the mobile apparatus 21 and then at block 473 a decision is madewhether or not the mobile apparatus 21 is located. If the mobileapparatus 21 was located in the selected camera zone of view, then theposition subroutine proceeds to tie point P2. If not, then at decisionblock 475, a decision is made as to whether or not all of the camerashave been searched. In the adverse case, then the navigation CPU andmemory 37a proceeds to increment "n" to the next camera zone at 477 andtie point P1 ties back into the input to block 469.

If all the members of the plurality of cameras 1 have been searched,then the rings of uncertainty 455 are incremented at block 479 and themobile apparatus 21 is labeled invisible at block 480 and then theroutine proceeds to the end of the subroutine via tie point P4.

Tie point P3 which is the "No" route taken after a decision is made atblock 458 provides for linearly scanning each camera zone of theplurality of cameras 1 once in search of the mobile apparatus 21 atblock 481. Then at block 483, a query is made, "was the mobile apparatus21 or robot located?". If the answer is no, then tie point P5 ties backinto labeling the mobile apparatus 21 invisible at block 480. Block 485,which is taken if the mobile apparatus 21 is located either at block 483or at block 473, labels the mobile apparatus 21 as being visible and thezone of uncertainty 455 is reduced to zero at block 487. The propercamera zone and the mobile apparatus's 21 position is identified atdecision block 489 and the subroutine ends at block 491.

Network for the Control of a Mobile Robot System

In FIG. 1, there is shown a mobile apparatus system for controlling asingle mobile apparatus 21. However, in many practical applications,there are more than one mobile apparatus 21 in a single factoryenvironment.

FIG. 34 illustrates the operation of a computer network for control of amobile apparatus system in which there are more than one mobileapparatus 21 operating. In FIG. 34, the terminal 17 interfaces with thecontrol and navigation circuit 15.

In particular, the terminal 17 interfaces to an operator interfacecontroller 501 which communicates to a communication processor 517. Alsoin communication with the communication processor 517 is a factorycontroller 503, a planner scheduler 505 and a navigation controller 507.Because the mobile apparatuses 21 are battery powered periodically theyneed to report in to a battery station for recharging. Therefore, therecan be a battery station controller 515 within the host CPU and memory37. Communications from the communication processor 517 communicate withthe controllers that are stored within each mobile apparatus 21 and inparticular in the CPU and memory 52. Therefore, there is a first mobileapparatus controller 509, a second mobile apparatus controller 511 andan nth. mobile apparatus controller 513.

FIG. 35 is a flow diagram of the operator interface 501 in which atstart position 1515 the operator interface waits and continuouslymonitors for any messages being received at 1517. If a message has beenreceived, then the message is processed and an update of the systemstatus is implemented at block 519. If no messages have been receivedfrom the communication processor at block 517, then a check is made tosee if any keyboard commands have been received at block 521. If nokeyboard messages have been received, then the update of the displaythat is on the terminal 17 is implemented at block 523. If a keyboardinput has been received then that information is processed at block 525and the operator interface controllers proceeds then to block 523 forupdating of the display on terminal 17. At block 527 the operatorinterface controller returns to the start position 1515 where itmonitors to see if any new status messages have arrived at block 1517.

FIG. 36 is the logic necessary to implement the communication processorprogram that is stored within the navigation CPU and memory 37a. At thestart block 529 the communication processor 517 is at rest and inputs amessage at block 531. If the message has multiple destinations, then atblock 533 this will be determined and the communication processor 517will proceed to block 535 to transmit the messages to all the requestedoutput queues or in the case where the message has only a singledestination then the message is placed in the appropriate queue at block537 and then the outgoing messages are transmitted at block 539. Thecommunication logic then returns to the start position at block 529 toinput a message.

The navigation logic 507 is implemented in FIG. 37. The mobile apparatus21 uses an onboard dead reckoning circuit to provide its navigation.However, periodically its absolute position is provided so that anydiscrepancies between the dead reckoning position and the actualposition may be compensated for. This process is implemented in FIG. 37where at block 543 the navigation controller 507 is at the startposition and it implements the search routine to recognize and locate amobile apparatus at block 545. Once the mobile apparatus 21 has beenlocated, then at block 547 its position is transmitted to the mobileapparatus 21 and the navigation controller 507 returns to the startposition at block 543.

FIG. 38 is the block diagram of the factory controller logic in whichthe factory control computer 503 is at rest at block 551. Decision block553 ascertains if any incoming material has arrived. If so, then acommand is generated and transmitted to the planner scheduler 505 totransport the material to a first processing station and a work order isgenerated at block 555. The factory controller monitors whether or not amobile apparatus 21 has arrived at block 557 at a designated stationregardless if any material is expected to arrive at block 553 or if thework order has been processed at block 555. If a mobile apparatus 21 hasarrived at a station, than at decision block 557, the station in whichthe mobile apparatus 21 has arrived is signaled to begin processing atblock 559. If not, or the processing has begun at block 559, the factorycontroller ascertains if any stations have indicated that their processis complete at block 561. If the process is complete, then the plannerscheduler 505 is commanded to transport the partially finished materialto the next sequential processing station or transport the fullyfinished material to the output dock where the work order is thencompleted at block 563 and the factory controller at block 565 returnsto the start position 551. If the process is not complete the factorycontroller takes the "no" route to block 565 from block 561.

A flow diagram for the planner scheduler 505 is provided in FIG. 39 inwhich at the start position 567 the planner scheduler 505 proceeds tocheck for any new work orders being received at decision block 569. Ifno new work order has been received, then the planner scheduler 505ascertains if any robots or mobile apparatuses 21 are available at block571. If a new work order is in the queue, then that work order isentered at block 573. The planner scheduler 505 then assigns additionalwork orders to the mobile apparatus 21 which can best do the work atblock 575 and the work plans are transmitted to the mobile apparatus 21at block 576 after which the planner scheduler 505 then returns to thestart position at block 577. The planner scheduler 505 continues tomonitor for work orders or the availability of mobile apparatuses 21 asindicated by path 579.

FIG. 40 is the logic for the battery station controller 515 in which atblock 579 the battery station controller 515 is at initial position andit monitors the batteries currently being recharged at block 581. Atdecision block 583, it ascertains if any new commands have been receivedand if so, then at block 585, it processes a new command to have thebatteries removed, inserted, recharged, etc. and returns to the startposition at block 587.

Controller for a Mobile Robot System

As was discussed in conjunction with FIG. 34, the planner scheduler 505schedules the movement of the plurality of mobile apparatuses 509,511-513. The system that is illustrated in FIG. 34 must coordinatetogether so as to minimize activities and distance traveled.

The planner scheduler 505 is implemented by the apparatus illustrated inFIG. 1 and includes a software program that is contained within the hostCPU and memory 37. In FIG. 41A there are six possible requests that theplanner scheduler 505 may implement when instructed to do so. Theserequests include, start up (block 601), material transfer (block 603),battery change (block 605), parking (block 607), location transfer(block 609), and transfer from system (block 611). The planner scheduler505 continuously cycles from the start block 600 through the six blockslooking for requests to implement.

Beginning with the rest or start position 600 the planner scheduler 505initiates a process in which at the first decision block 601 the plannerscheduler 505 ascertains if there is a request for a start up of amobile apparatus 21 such as a robot. If there is a request or a "yes"answer then subroutine A is taken on FIG. 41B and if there is not arequest for a mobile apparatus 21 start up or a "no" answer then theplanner scheduler 505 proceeds to the next decision block at 603.

Decision block 603 ascertains if a request for material transfer hasbeen received. If a request has been received, then the "yes" line istaken and the planner scheduler 505 proceeds to tie point B located inFIG. 41C. In the event no request has been received, then the no line istaken and the planner scheduler 505 proceeds to block 605 whichascertains if a request for battery change has been received. If arequest for a battery change has been received, then the yes line istaken and the subroutine C indicated in FIG. 41D is implemented or ifthe no line is taken (no request), the planner scheduler 505 proceeds todecision block 607.

Decision block 607 ascertains if a request to transfer the mobileapparatus 21 to a parking location has been received. Of course if ithas, the "yes" path taken and the subroutine D illustrated in FIG. 41Eis implemented and if not the "no" path is taken and the plannerscheduler 505 proceeds to decision block 609. At decision block 609 theplanner scheduler 505 ascertains if a request has been received totransfer the mobile apparatus 21 to another factory location. If "yes",then the subroutine at tie point E is implemented on FIG. 41F. If "no"the planner scheduler 505 proceeds to decision block 611.

The mobile apparatus 21 may be removed from the manufacturingenvironment at decision block 611 where a decision is made if there hasbeen a request to remove the mobile apparatus 21 from the floor of themanufacturing facility. If "no", then the planner scheduler 505 cyclesback to the start position 600. If "yes" the planner scheduler 505proceeds to tie point F implemented on FIG. 41G.

It can be seen that this program can include a plurality of multiplerequests that the planner scheduler 505 can implement by stringingtogether the decision blocks and then implementing the routines toexecute those decisions.

At tie point A in FIG. 41B, the planner scheduler 505 initiatescommunications with the mobile apparatus 21 at block 613. Thiscommunication, of course, is conducted by the communication processor517 of FIG. 34. If the addressed mobile apparatus 21 fails to respond,then at decision block 615 a decision made to go back to the startposition 613 meaning there is no response and to reinitiate thecommunications with the mobile apparatus 21. After a certain number oftries at block 64 then the planner scheduler 505 will return to thestart position at block 600 to verify if another request has beenreceived. If the mobile apparatus 21 responds, then at block 617 thenavigation CPU and memory 37A is commanded to begin to track thecontrollable mobile apparatus 21. A determination is made of thelocation upon the floor of the facility in which the controllable mobileapparatus 21 is used at block 619 and then at block 621 the controllablemobile apparatus 21 is moved to the closest available destination point.At the completion of block 621, the planner scheduler 505 returns toblock 603 for the next decision.

At block 603 a decision is made to ascertain if a request for a materialtransfer has been received. If it has, then the planner scheduler 505implements the material transfer subroutine via tie point B. In FIG.41C, the subroutine implemented at tie point B includes an attempt toschedule a material transfer at block 623. If the planner scheduler 505ascertains that it is not possible to attempt a transfer, then theplanner scheduler 505 returns to block 605 and proceeds through the loopthat is illustrated in FIG. 41A. If a material transfer is possible,then a path is generated through the factory for the controllable mobileapparatus 21 to follow at block 625. The mobile apparatus 21 isinstructed to move through the manufacturing facility at block 627,according to the path information that is provided by the plannerscheduler 505 and will do so according to its navigation generationprogram includiing its dead reckoning program that was discussed inconjunction with FIGS. 18 through 22. Upon reaching its finaldestination, the factory controller 503 and the operator interfacecontroller 501 are notified by the planner scheduler 505 that the mobileapparatus 21 has reached its final position at block 629.

A decision is made at block 631 to ascertain if the material transfermechanism is on board the mobile apparatus 21. If it is, then thematerial is transferred at block 633. If not, at block 635 the unitwaits for the material transfer to complete after which the subroutineis finished and the process returns to tie point B until all thematerial has been transferred after which it returns to the block 605 ofFIG. 41A.

Tie point C is implemented in FIG. 41D, and it is only used when thereis a battery changing apparatus available and includes a decision block637 to ascertain that a battery changer is available and if so, it isavailable for use. If the battery changer is unavailable, then theplanner scheduler 505 returns to block 607 and proceeds through the loopthat is illustrated in FIG. 41A. If it is, then the planner scheduler505 at block 639 generates a path through the manufacturing facility forthe mobile apparatus 21 to follow.

The mobile apparatus 21 navigates through the manufacturing facilityaccording to the path information at block 641. The planner scheduler505 notifies the battery changing apparatus at block 643 that the mobileapparatus 21 has reached its final destination. The battery is thenchanged at block 645 and the planner scheduler 505 returns to block 607and proceeds through the loop that is illustrated in FIG. 41A. Thesubroutine that is connected to tie point D is illustrated in FIG. 41Eto which reference should now be made. It implements the parking of thecontrollable mobile apparatus 21. Block 647 ascertains if there are anyparking places available for use. If not, then the subroutine returns toblock 609 in FIG. 41A. If there is, then a path is generated at block649 and the mobile apparatus 21 is moved through the manufacturingfacility to the parking space at block 651 after which the subroutinereturns to block 609 in FIG. 41A. FIG. 41F illustrates a subroutine thatimplements a request to send a mobile apparatus 21 to another locationwithin the manufacturing facility. If this route is taken, then as shownin FIG. 41F, a decision is made to ascertain of a destination isavailable for use at decision block 653. If it is not, the subroutinereturns to block 611 in FIG. 41A. If it is, a path is generated at block655 and the mobile apparatus 21 is instructed to move there and providethe notification of its completion of the move at block 657.

Subroutine F exits the mobile apparatus 21 from the manufacturingfacility. It must be determined as shown at decision block 659 of FIG.41G is the exit is available. If it is not available, the subroutinereturns to block 601 in FIG. 41A. If it is, then a path is generated atblock 661 and the mobile apparatus 21 is instructed to move and notifythe planner scheduler 505 of the completion of this move at block 663.

Coordinating the Interaction of Multiple Robots

FIG. 42 to which reference should now be made is a schematic floor planof a manufacturing facility 700. Points 700 through 708 are possibledestination points for the mobile apparatus 21 and hereinafter will bereferred to as nodes. Lines 709 through 716 are permissible pathsegments between the nodes 701 through 708 along which the controllablemobile apparatus 21 may travel in the direction indicated by the arrows.All other regions of the manufacturing facility 700 are off limits tothe controllable mobile apparatus 21. Boxes 717 through 720 arelocations within the manufacturing facility 700 where the mobileapparatus 21 will perform a service. The mobile apparatus 21 may travelfrom one location to another location provided a set of continuous pathsegments exist between the two locations.

When multiple mobile apparatuses 21 are allowed to travel along the samepath, each mobile apparatus's 21 arrival at a node 701 through 708 mustbe coordinated in time with the arrival and departure of other mobileapparatus 21 to avoid deadlocks and collisions. To coordinate the mobileapparatuses' 21 travel through the manufacturing facility 700, a set ofrules which control the arrival and departure of the mobile apparatus 21at each node along a defined path are generated. In the event multiplemobile apparatuses 21 must visit the same node, then these rulesdetermine which mobile apparatus 21 is allowed to visit the node first.Once a set of rules is generated, then these rules are followed until amobile apparatus 21 visits a node which will not be occupied by anothermobile apparatus 21.

FIGS. 43 through 48 are used to illustrate the creation of rules whichcontrol the arrival and departure of the mobile apparatuses 21. FIGS.43, 45 and 47 show the path of the mobile apparatus 21 which in the caseof the figures are apparatuses R1, R2 and R3 and FIGS. 44, 46 and 48show the nodes where the apparatuese R1, R2 and R3 will be for aspecific time interval.

Referring to FIG. 43 is a set of path segments for a plurality of mobileapparatuses 21 designated R1, R2 and R3 to follow through a typicalmanufacturing facility 700. Point 721 through 728 are destination nodesand lines 729 through 736 are permissible path segments between thenodes. In FIG. 43 only the path 737 for the mobile apparatus 21 that isdesignated as R1 has been generated. The mobile apparatus 21 will startmoving from node 721, will pass through the node 723, 724, and 726 andthen stop at node 727. The nodal position of the mobile apparatus 21that is designated R1 for each of the six time invervals is indicated inFIG. 44.

In FIG. 44 the rules that control the movements of the mobileapparatuses R1, R2 and R3 are defined. Because the apparatus designatedR1 for apparatus 1 will not visit any node where another mobileapparatus 21 will visit, there are not rules set for the path at thistime. In FIG. 43, however, as in the embodiment of FIGS. 45 and 47 theapparatus R1's first node is 721 and its last node is 727.

In FIGS. 45 and 46 the mobile apparatus 21 as designated Robot 1 or R1is as the node 721 and it is to travel to 727. The path nodes that itwill traverse on this route to node 727 includes the starting node 721and nodes 723, 724, 726 and 727. Additionally, in the embodiment of FIG.45, the second mobile apparatus 21 that is designated R2 is at astarting node 722 and its last node is node 728 and it is to travel thepath 738. This path includes the nodes 722, 723, 724, 726 and 728.

In FIG. 46 the nodal positions are illustrated with respect to time forboth mobile apparatuses 21 that are designated R1 and R2 respectfully.FIG. 46 also provides the rules that coordinate each apparatus's R1 andR2 visit to the nodes in the manufacturing facility 700. As was shown inFIG. 43, the mobile apparatus 21 that is designated R1 has alreadyselected a path through nodes 723, 724 and 726. Therefore, the secondmobile apparatus 21 as designated R2 has to move from node 722 to node728. A path is generated without changing the path for R1.

During the first time interval R2 will attempt to move to node 723.However, R1 has priority on the occupation of node 723 and will movethere during the second time interval. Therefore, the rules are that R1must arrive at node 723 before R2 and R2 must arrive after R1. Duringthe second time interval, R2 must wait before moving to node 723. R2 canoccupy node 723 in the third time interval. Because R2 must arrive atnode 723 after R1, the rules for R2 change to arrive after departing ofR1. The rules for R1 also change to arrive before the arrival of R2.Because the rules have now been set, they are applied to nodes 724 and726. However, there are no conflicts between R1 and R2 at nodes 727 and728 thus the rules no longer apply.

FIGS. 47 and 48 illustrate the paths for a third mobile apparatus 21 orR3 with the previous paths for the controllable mobile apparatuses 21that are designated R1 and R2. R3 will move from node 728 along path 739and pass through nodes 726, 725 and 723, stopping finally at node 722.Nodes 726 and 723 are common to all three mobile apparatuses 21 that aredesignated in FIG. 47 as R1, R2 and R3.

FIG. 48 shows the nodal positions with respect to time for all threemobile apparatus's 21. In FIG. 48 the rules which coordinate eachapparatus 21 visits to the nodes in the manufacturing facility areillustrated. As shown in FIG. 46, R1 and R2 have already selected a paththrough nodes 723 and 726, therefore, for R3 to move from node 728 tonode 722 a new path is generated without changing the previous paths forR1 and R2.

R3 will first move to node 726. In view of the fact that neither R1 norR2 occupy node 726 during the second time interval R3 will be allowed tooccupy node 726. R3 will arrive at node 726 before the other mobileapparatuses 21 and therefore the rules for R3 are defined as arrivalbefore R1. The rules, of course, for R1 and R2 will not change.

The third time interval will be the time for R3 to move to node 724 ornode 725. The earliest possible time that R3 can occupy node 724 is atthe fifth time interval due to the prior establishment of paths thatwere discussed in conjunction with FIGS. 45 and 46. R3 must arrive atnode 724 after R1. The rules further state that R3 must arrive at allnodes before R1, therefore, the rules will be violated for R3 to move tonode 724. Because neither R1 nor R2 will occupy node 725 at any time, R3can occupy node 725 during the third time interval.

When one of the plurality of mobile apparatuses 21 that are in theexample of FIG. 47 designated R1, R2 or R3 visit a node that will not beoccupied by another mobile apparatus 21, the rules for the mobileapparatus 21 are reset, therefore R3 has no rules to control itsmovement to node 725. R3 then will move to node 723 at the fourth timeinterval. R3 will arrive at node 723 after R1 and R2 have departed andtherefore the rule for R3 is to arrive after the departure of R2.Because the rules have been set for R3 they are also applicable to node722. Thus, by generation of rules based upon each move of a mobileapparatus 21, no mobile apparatus 21 will occupy a node at the same timethat a second mobile apparatus 21 is destined to occupy the node.

FIGS. 49 provide a flow diagram for implementing the coordination of amultiple apparatus system in a manufacturing facility as was discussedin conjunction with FIGS. 42-48. In particular, in FIG. 49A the startposition is at point 750. Leaving point 750 there is a first decisionblock, block 751, which decides if there are more nodes connected by apath segment to the current node If the answer is "no", then tie point His used and connects over to FIG. 49C. If the answer is "yes", theplanner scheduler 505 will retrieve the next node connected to the pathsegment at block 755 and at block 753 will compute the arrival anddeparture time at the next node. The program then proceeds to block 757where the arrival time at the next node is compared to determine if itis greater than the lowest travel time to the final destination. If theanswer is "no", tie point C is taken to FIG. 49, if the answer is "yes",then tie poing H is taken to FIG. 49C.

Following the "no" path and proceeding to FIG. 49 at decision block 758a query ascertains, "Will the current mobile apparatus 21 arrive at thenext node after another mobile apparatus 21 departs from the node?". Ifthe answer is "no", then tie point D is taking to FIG. 49C at the startof block 763. If the answer is "yes" then the next stop in the sequenceis, "According to the predefined rules should the current mobileapparatus 21 arrive at the next node before the other mobile apparatus21 arrives at that node (block 759)?". If the answer is "yes", then tiepoint H which is located on FIG. 49G is taken. However, taking the "no"path, then at block 761 the visit rules for the next nodes aregenerated. The rules state that the current mobile apparatus 21 mustarrive at the next node after the other mobile apparatus 21 departs fromthat node, then proceeding to tie point F which is located on FIG. 49E.However, FIG. 49C is the D path that is taken from block 758 and will bediscussed at this time.

At the first decision block, (block 763) a query ascertains, "Will thecurrent mobile apparatus arrive at the next node before the other mobileapparatus departs from that node?". If the answer is "no", then the pathvia tie point E is taken which goes to FIG. 49D. If the answer is "yes",then the planner scheduler 505 proceeds to block 765 which makes adecision, "According to the rules should the current mobile apparatusarrive at the next node after the other mobile apparatus departs thatnode?". The "yes" path will increase the current arrival time at block767 and then proceeds to tie point B which is prior to block 757 on FIG.49A. If the answer is "no", then the decision is made, "Is the othermobile apparatus currently located at the next node?", at decision block769. The "yes" path will result in the program jumping to implement theblocks that follow the path H that is illustrated in FIG. 49G. If theanswer is "no", then the planner scheduler 505 will generate the visitrules for the next node at block 781.

The rules state at block 781 that the current mobile apparatus mustarrive at the next node before the other mobile apparatus departs thatnode and then the program proceeds to subroutine F which is located inFIG. 49E.

The E subroutine or E path from block 763 is illustrated in FIG. 49D andafter it has been determined that the current mobile apparatus will notarrive at the node before another mobile apparatus departs that nodethen at block 783 a decision is made that according to the rules shouldthe current mobile apparatus arrive at the next node before anothermobile apparatus arrives at that node. If the answer is "no", then thecurrent mobile apparatus' arrival time is incremented at block 785 andit proceeds to tie point B which is located in FIG. 49A just prior todecision block 757. If the answer is "yes" then tie point H is takenwhich is in FIG. 49G.

After the completion of block 781 of FIG. 49C, the planner scheduler 505takes path F which is illustrated in FIG. 49E and has a first decisionblock at 787 which ascertains whether all the mobile apparatuses havebeen checked for conflicts at the next node. If the answer is "no", thenpath C is taken which is illustrated in FIG. 49B and will continue thegeneration of rules for resolving of conflicts. If the answer is "yes",then the planner scheduler 505 will proceed to block 788 whichdetermines of any mobile apparatus will visit the next node. If theanswer is "no", then the rules for the current mobile apparatus at thenext node are reset. If the answer is "yes", then the program proceedsto path B which is illustrated in FIG. 49A.

However, taking the "no" path, the rules are reset for the next node atblock 789 and then at block 790 all the node visit parameters are savedwhich were generated for the next node and then the next node is now thecurrent node at block 791.

The program then proceeds to tie point G which is illustrated in FIG.49F which at block 792 makes a decision, "Is the next node the finaldestination for the mobile apparatus?". If the answer is "no", theprogram proceeds to the start of block 751 that is illustrated on FIG.49A. If the answer is "yes", "Is the current arival time the lowestarrival time?" is then asked at block 793. If the answer is "no", theprogram proceeds to block H, which is illustrated in FIG. 49G. If theanswer is "yes", then all the parameters are saved for the node which isbeing visited and for all the nodes in the current path at block 794.The planner scheduler 505 proceeds to tie point H which is illustratedin FIG. 49G.

FIG. 49G makes a decision at block 795, "Are any more path assignmentsdefined for the current node?". If the answer is "no", then the programproceeds to tie point A on FIG. 49A which is essentially the startposition. If the answer is "yes", then it proceeds to block 797 where itimplements a step of backing up one node in the current path. Atdecision block 799 a decision is made, "Is this node the first node inthe path?". If the answer is "yes", then it proceeds to step 800. If theanswer is "no", then it proceeds to tie point A which is prior to block751, FIG. 49A.

Taking the "yes" path from block 799 to block 800 which questions, "Wasa possible path identified for the mobile apparatus 21 to follow?", theyes path will cause the unit to proceed to block 801. If the answer is"no", the no path will cause the system to exit at block 803. All thenode visit parameters which were saved into the final path table arecopied at block 801 and after which the commands are generated to directthe mobile apparatus 21 to move along the final path.

Controlling Movements of Robots in a Multi-Node environment

As was discussed in conjunction with FIG. 34, the planner scheduler 505is responsible for generating move commands to the individualapparatuses 509 through 513, however many are being used in themanufacturing facility 700. The communication processor 517 isresponsible for sending move commands to the mobile apparatuses 21 overthe wireless communication system 19 of FIG. 1. Each CPU and memory 52that is contained within each controllable mobile apparatus 21 isresponsible for executing the move commands.

As was discussed in conjunction with FIG. 42 which is a schematic of atypical manufacturing floor plan, nodes 701 through 708 are possibledestination points for the mobile apparatus 21 along paths 709 through716. The arrows on the lines indicate the possible directions of travelfor the mobile apparatus 21. All other regions of the manufacturingfacility 700 are designated as being off limits. To move a mobileapparatus 21 through the manufacturing facility 700 the plannerscheduler 505 must first allocate nodes from a previously generated pathto a mobile apparatus 21 and then command the mobile apparatus 21 tomove to the nodes. At any one time the mobile apparatus 21 may be movedinto a node and have several subsequent move commands buffered in itsmemory that is a part of CPU and memory 52. As a mobile apparatus 21moves from node to node, the mobile apparatus 21 informs the plannerscheduler 505 of each node it has passed. When the planner scheduler 505has been informed that a node has been passed, the node is thendeallocated and becomes available for other mobile apparatus 21.

FIGS. 50 illustrate the a implementation of the above discussed processand FIG. 50A is a flow diagram of the transmission of instructions andcommands between the planner scheduler 505 and mobile apparatus 21. Theprocess begins at block 805. Decision block 806 ascertains if the mobileapparatus 21 has arrived at the final destination node. If it has thenthe exit is taken at block 809. If it hasn't then the planner scheduler505 proceeds to block 807 where a decision is made that determines ifthe maximum number of nodes are already allocated to the mobileapparatus 21.

If the maximum number of nodes has been allocated, then path A is takento FIG. 50B which at block 815 waits for the mobile apparatus 21 tocomplete the next move. The system at block 817 deallocates all nodeswhich the mobile apparatus 21 has passed and returns via tie point 819to FIG. 50A.

At block 808 of FIG. 50A the query, "Can the next node be allocated tothe mobile apparatus 21?" is asked. If the answer is "no" then tie pointB is taken to FIG. 50C where at block 818 the system waits for the nodeto be deallocated by another mobile apparatus. At the completion of thestep at block 818 the system returns to FIG. 50A via tie point 821.

At block 811 the mobile apparatus's 21 trajectory for the next segmentis determined as is illustrated in FIG. 42. Then at block 813 the movecommand is transmitted to the mobile apparatus 21 to be executed. Thisprocess continues until the mobile apparatus 21 arrives at the finaldestination node as is defined in block 806.

The Mobile Robot's Executive Program

There are two major sequences that are implemented by the CPU and memory52 that is contained within each mobile apparatus 21. These are theexecutive and control command structures. The executive commandstructure runs continuously to provide system services support forcommunications, options and command decoding and formatting of messagesthat are transmitted between each mobile apparatus 21 and thecommunication processor 517. The control sequence is interrupt drivenand is responsible for the dead reckoning, navigation and controlsystems that have been previously discussed. The mobile apparatus 21movement through a manufacturing facility 700 utilizes all of thesesystems. A command from the host CPU and memory 37 is transmitted viathe communication system 19 to the mobile apparatus 21. The executivesequence receives and decodes these commands. For a move command, it isformatted and placed in the execution queue for the control sequence toexecute. The executive sequence then checks for a change in the state ofthe mobile apparatus 21 caused by a command completion or a change inthe status of the mobile apparatus 21. If necessary, the executive willtransmit a message to the navigation CPU and memory 37 detailing thestate change before returning to receive another command from the hostCPU and memory 37. This loop is executed continuously.

When a timer interrupt occurs, the control program sequence will commandthe steering and drive servo for each move. It then calculates thecurrent mobile apparatus 21 position using the data from the control andnavigation circuit 15 which was previously discussed and performs thedead reckoning and navigation requirements.

Depending on its current position and state it is possible that thecurrently executing command has been completed. If so, the next commandin the queue is selected for execution. The executive program notes thisface and builds the state change message for transmission to the hostCPU and memory 37. The mobile apparatus 21 control software then becomesidle waiting for the next timer interrupt.

Referring to FIG. 51 which describes the executive program which atblock 850 is continuously monitoring for a message being received overthe communication system 19. If no message is being received, then path827 is taken and if a message is being received, then the CPU and memory52 proceeds to block 826 where the message is placed on the mobileapparatus 21 queue if required. Options support is then provided atblock 829. Decision block 831 decides whether or not a command pointerhas changed. If "not", then the program proceeds to block 850. If thecommand pointer has changed then the mobile apparatus 21 informs thehost CPU and memory 37 of a command completion and mobile apparatus 21position at block 833. At the completion of the step indicated at block833, the program returns to the input of block 850.

The Mobile Robot's Control Program

As was discussed previously, the control program sequence is implementedin FIG. 52 where at block 835 each of the steering and drive servos arecommanded. The mobile apparatus 21 position is calculated at block 837,this was previously discussed, and the dead reckonong commands aregenerated at block 839. Decision block 841 determines if the commandpointer or path needs to be updated. If it does, then it is updated atblock 843, and if it doesn't the control program proceeds to additionalprocessing to build a status table for placing and transferring to themobile apparatus 21 executive sequence at block 845. Which, by the way,is also implemented if there is no need to update the command positionvia path 847. The control program then proceeds back to the input ofblock 835 via path 849.

Navigation System for Multiple Robots

In the complex system of multple mobile apparatuses 21 moving across themanufacturing facility as is illustrated in FIG. 42, the navigation CPUand memory 37a is continuously predicting the mobile apparatus thatneeds a position fix. Referring to FIG. 53A, at block 851 the mobileapparatus with the greatest need for a position fix is identified. Amessage is broadcast to all mobile apparatus via the communicationprocessor 517 to turn on the proper navigation beacons and turn off allother beacons and suspend all communications at block 853. The next stepinputs all the messages from the communication processor at block 855. Adecision is made at decision block 857 if the message is a navigationacknowledgement. If it is not, then the message is processed at block859. If it is, then the control and vision navigation circuit 15 acquirea video image at block 860 after which a release is transmitted to themobile apparatuses 21 to continue their infrared communications at block861.

The image from the selected member of the plurality of cameras 1 isanalyzed at block 863. After the analysis of the input image from theselected camera 1S, the navigation CPU and memory 37a proceeds via tiepoint P2 to block 864 of FIG. 53B where the mobile apparatus' 21 imageis analyzed and ascertained if it is recognized at block 864. Thisprocess has been previously discussed in conjunction with the controland navigation circuit 15. If the answer is "no" then a decision is madeto ascertain if all the cameras that are members of the plurality ofcameras 1 have been searched at block 865. If that answer is "no", thena prediction is made for the next most likely camera at block 867 andthe program proceeds to tie point P3 which is on FIG. 53A just prior toimplementing the step contained in block 853.

If all the likely cameras have been searched, then tie point P1 is takenjust prior to block 851. If the apparatus's image is recognized, thenits position and heading is ascertained at block 864 and this positionis transmitted to the mobile apparatus 21 at block 866 after which theunit returns to the tie point P1 for the start of the process.

The Communication Processor

The communication processor 517 and in particular the interface isimplemented in FIG. 54A which should be used in conjunction with FIGS. 1and 34. At block 510 the communication processor 517 will start theprocess and wait for a navigation request from the navigation controller507. The communication processor 517 enters the navigation request(block 868) and broadcasts the message into the output queue fortransmission via the wireless communication system 19 (block 869). Thecommunication processor 517 transmits the outgoing message via awireless communication system such as an infrared system until itrecognizes the navigation, "NAV", request message at block 870. Thecommunication processor 517 broadcasts the "NAV" request to all themobile apparatus controllers 509 through 513 and sends a navigationacknowledgement to the navigation controller 507 and shuts offcommunications to all mobile apparatus controllers at step 871. Thecommunication processor continues to queue all incoming messages butdoes not send any messages via the wireless communication link at block871.

At block 872 the communication processor 517 waits for a synchronizationmessage from the navigation controller 507. The communication processor517 in FIG. 54B then recognizes the synchronization message and enablesthe communications system 19 to transmit all messages in its outputqueue at block 873. At block 874 the position fix message is receivedfrom the navigation controller 507 and placed in the communicationoutput queue connecting to the communication system 19. The position fixmessages are transmitted to mobile apparatus controllers 509 through 513at block 875 and the program then returns to the start position at block510.

The Message Output Process of the Communication Processor

The communication output process is illustrated in FIG. 55 where atblock 876 the output queue is stopped from emptying its messages i.e.the output port is halted. The message is sent to the mobile apparatus(device) at block 877 and at block 878 the communication processor 517waits for a navigation request. If none is received it returns to thestart of block 876. If one is received, it halts the communicationoutput process at block 879 and then proceeds to the port halted blockat block 876.

The Message Input Process of the Communication Processor

In FIG. 56 the communication processor 517 monitors its input queue forthe receipt of messages at block 881. It ascertains whether a message isfrom the navigation controller 507 at block 882. If it is "not", it putsthe message in the correct output queue at block 883 then returns toblock 881. It it is, it checks to see if the message is a navigationsynchronization message at block 884. If it is not, the communicationprocessor 517 proceeds to block 883. If it is, it restarts the mobileapparatus output process at block 885 and proceeds to block 883.

Collision Avoidance

When the mobile apparatus 21 moves along its path in a manufacturingfacility 700, a certain amount of deviation due to the dead reckoningsystem from a commanded path can be expected and is tolerated. As theplurality of mobile apparatus controllers 509 through 513 move in asystem, a window is projected around the current path of each mobileapparatus 21. FIG. 57 illustrates the first step in building the windowaround a mobile apparatus's 21 path. The window, between points 891 and887 is along the mobile apparatus's 21 path and is used to define astraight line 893 that the mobile apparatus's 21 path should follow. Themobile apparatus's 21 allowable deviation from the straight line 893 isdefined by a special distance as indicated by dimension line 894.

FIG. 58 illustrates the second step in building the window around themobile apparatus's 21 path. The distance from the straight line 893defines the size of the window but does not control the distance themobile apparatus 21 may travel from the end points of the line 893. Thedistance from each end point is used to complete close the window. Aradius 901 of a circle 1200 formed around each end point 891 and 887 isthe distance of the path segment 893 plus a constant 904 to allow fordeviations in the path of the mobile apparatuses 21.

In FIG. 59 the shaded area 905 is the mobile apparatus's 21 window thatis established by dimension lines 894 plus the two circles drawn aroundthe end points 887 and 891. If the mobile apparatus's 21 position is notwithin the window, then the mobile apparatus 21 is stopped.

FIG. 60 illustrates a control sequence for implementing this function.The system begins at block 907 and proceeds to block 909 where it waitsfor the mobile apparatus' 21 most current position. At block 911, thedistance between the mobile apparatus 21 and the line between the endpoints is ascertained and then a decision at block 981 is made basedupon the visual position of the mobile apparatus 21 and if that positionis within the partial window of FIG. 57.

If the mobile apparatus 21 is not within the partial window of FIG. 57,then the mobile apparatus is stopped at block 917 and the controlsequence returns to the start positions.

If the mobile apparatus 21 is within the partial window, then thedistance between the mobile apparatus 21 and each end point is computedat block 913. At block 915 a decision is made to determine of the mobileapparatus 21 is within the computed distance. If it is, then line 918 istaken back to the start position.

The distance of the mobile apparatus 21 from line 893 of FIG. 57 isdetermined by equation 4 wherein R_(x) is equal to the x coordinate ofthe mobile apparatus 21, R_(y) is equal the the y coordinate of themobile apparatus 21, P_(x) is equal to the x coordinate of the endpoint, P_(y) is equal to the y coordinate of the end point. V_(x) isequal to the distance between the end points along the x axis and V_(y)is equal to the distance between the end points along the y axis.Equation 5 is used to calculate the radius that is generated around eachend point. The radius is equal to the square root of (V_(x) ² +V_(y)²)+the constant 904. Both equations 4 and 5 are in the table ofequations.

I claim:
 1. A system for predicting tracking cameras used for navigatinga mobile apparatus that is electrically controllable, comprising:aplurality of independent, selectable cameras, each having apredetermined field of view of an area used for travel by the mobileapparatus; associated with the mobile apparatus, a plurality of lightedpoints arranged in a predefined, identifiable pattern detectable by saidcameras; control and navigation means for selecting the video output ofone at a time of said cameras to determine the location of the mobileapparatus by using the video output of one of said cameras in whosefield of view the mobile apparatus is to be found, wherein said camerasare in electrical communication with said control and navigation means;wherein said control and navigation means includes a computer havingmemory for storing predetermined instructions for finding said one ofsaid cameras in whose field of view the mobile apparatus is to be found.2. The system of claim 1 wherein said computer includes:means forstoring the last known position of the mobile apparatus as it leaves thefield of view of one of said cameras.
 3. The system of claim 2 whereinsaid computer further includes:means for determining a radius ofuncertainty, centered upon said stored last known position, wherein saidradius is at least equal to the distance that the mobile apparatus couldhave traveled moving at its maximum speed during the period of timeelapsed after having been at said last known position.
 4. The system ofclaim 3 wherein said computer further includes:means for determiningwhether the area within a circle with its center at said last knownposition and its radius equal to said radius of uncertainty, overlapsany areas within the field of view of any of said cameras.
 5. The systemof claim 4 wherein said computer further includes:means for effectingthe selection, one at a time, of any camera whose field of view overlapssaid area within said circle, to search said field of view for themobile apparatus.
 6. The system of claim 5 wherein said computer furtherincludes:means for increasing the radius of uncertainty if the mobileapparatus is not found within the field of view of any of said cameraswhose field of view overlaps said area within said circle.
 7. A methodof selecting navigation cameras used for finding the position of amobile apparatus, wherein the selection is made from among a pluralityof independent, selectable cameras, each having a field of view of anarea traveled by the mobile apparatus, wherein the position of themobile apparatus can be determined from the video output of one of saidcameras in whose field of view the mobile apparatus is found, comprisingthe steps of:tracking the mobile apparatus in the field of view of oneof said selectable cameras; saving the last known position of the mobileapparatus when it leaves the field of view of said one of saidselectable cameras; calculating the maximum distance that the mobileapparatus could have traveled if it was traveling at its maximum speedafter leaving said field of view of one of said selectable cameras;choosing a radius of uncertainty at least as large as said maximumdistance; selecting only those cameras in whose field of view the mobileapparatus could be found it is had traveled a straight line distanceequal to or less than said radius of uncertainty, in any straight linefrom said last known position; and examining the video output of as manyof said selected cameras as is necessary to locate the mobile apparatus,or until the video output of all said selected cameras has beenexamined, whichever occurs first.
 8. The method of claim 7, furthercomprising the step of increasing the size of the radius of uncertaintyif the mobile apparatus was not located, and re-executing said steps ofselecting cameras and examining the video output, until all of saidplurality of cameras has been selected and their video outputs examined.9. The method of claim 8, wherein all of said steps are carried out on acomputer.
 10. A computerized method of navigating a mobile apparatususing a plurality of independent cameras whose fields of view coverareas traveled by the mobile apparatus, comprising the stepsof:determining the position of the mobile apparatus from the videooutput of a first camera in whose field of view the mobile apparatus isfound; storing the last position determined using the first camera asthe mobile apparatus leaves the field of view of the first camera;calculating a radius of uncertainty; selecting all cameras whose fieldof view intersects the area enclosed by the circle centered at said lastposition and whose radius is equal to said radius of uncertainty; andexamining the video output from each of said selected cameras one-by-oneuntil the mobile apparatus is relocated in a second camera's field ofview, or until all of said selected cameras' videw output has beenexamined, whichever comes first.
 11. The computerized method of claim10, further comprising the steps of:if the mobile apparatus wasrelocated, determine the position of the mobile apparatus using saidsecond camera; otherwise if the mobile apparatus was not relocated,increase the radius of uncertainty and re-execute all steps subsequentto said step of calculating a radius of uncertainty.
 12. Thecomputerized method of claim 11, wherein the step of increasing theradius of uncertainty is not executed if all said plurality of camerashave already been selected.
 13. The computerized method of claim 10,wherein said step of calculating a radius of uncertainty comprises thesteps of:calculating the maximum distance that the mobile apparatuscould have traveled since it was at said stored last position, if it hastraveled at its maximum possible speed; and making the radius ofuncertainty at least as large as said maximum distance.
 14. Thecomputerized method of claim 10, wherein said plurality of cameras is aplurality of CCD cameras.
 15. The computerized method of claim 10,wherein said position of the mobile apparatus is determined as an X-Yposition with respect to a factory floor coordinate system.